{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mI6aVBQRy6LC"
      },
      "source": [
        "# 4️⃣ Zero-Shot Cross-Lingual Transfer using Adapters\n",
        "\n",
        "Beyond AdapterFusion, which we trained in [the previous notebook](https://github.com/Adapter-Hub/adapters/blob/master/notebooks/04_Cross_Lingual_Transfer.ipynb), we can compose adapters for zero-shot cross-lingual transfer between tasks. We will use the stacked adapter setup presented in **MAD-X** ([Pfeiffer et al., 2020](https://arxiv.org/pdf/2005.00052.pdf)) for this purpose.\n",
        "\n",
        "In this example, the base model is a pre-trained multilingual **XLM-R** (`xlm-roberta-base`) ([Conneau et al., 2019](https://arxiv.org/pdf/1911.02116.pdf)) model. Additionally, two types of adapters, language adapters and task adapters, are used. Here's how the MAD-X process works in detail:\n",
        "\n",
        "1. Train language adapters for the source and target language on a language modeling task. In this notebook, we won't train them ourselves but use [pre-trained language adapters from the Hub](https://adapterhub.ml/explore/text_lang/).\n",
        "2. Train a task adapter on the target task dataset. This task adapter is **stacked** upon the previously trained language adapter. During this step, only the weights of the task adapter are updated.\n",
        "3. Perform zero-shot cross-lingual transfer. In this last step, we simply replace the source language adapter with the target language adapter while keeping the stacked task adapter.\n",
        "\n",
        "Now to our concrete example: we select **XCOPA** ([Ponti et al., 2020](https://ducdauge.github.io/files/xcopa.pdf)), a multilingual extension of the **COPA** commonsence reasoning dataset ([Roemmele et al., 2011](https://people.ict.usc.edu/~gordon/publications/AAAI-SPRING11A.PDF)) as our target task. The setup is trained on the original **English** dataset and then transferred to **Chinese**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3L9gYpCV28OA"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Besides `adapters`, we use HuggingFace's `datasets` library for loading the data. So let's install both first:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:24:50.921564600Z",
          "start_time": "2023-08-17T09:24:44.078877400Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qL3Sq1HQynCq",
        "outputId": "aa36636a-e484-4ca8-bc6e-b54b37b098b7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m204.3/204.3 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m61.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m25.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m104.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m48.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m519.3/519.3 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m19.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m251.2/251.2 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -Uq adapters\n",
        "!pip install -Uq datasets\n",
        "!pip install -Uq accelerate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fP9OMSUT-FtL"
      },
      "source": [
        "## Dataset Preprocessing\n",
        "\n",
        "We need the English COPA dataset for training our task adapter. It is part of the SuperGLUE benchmark and can be loaded via `datasets` using one line of code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:25:09.494858700Z",
          "start_time": "2023-08-17T09:25:06.870881400Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258,
          "referenced_widgets": [
            "660b81766f7044b6a1ce2f8c2c45f461",
            "1250b158a5b945da8d19b0ecdfefde34",
            "6b2003198e274ef7a474ec222624be1c",
            "8ee881038ff4488382ca006a07c9ed6d",
            "4fa6d2b22bbe4c1a87cb436dfecefe0c",
            "0ac38ac59979437196a82149dab2fb14",
            "7815c3cab73740b29687228f3e0d1fa7",
            "38411915569f487c9bc36f29c292935e",
            "c601dba9e79840aeb7f832732564128c",
            "0781ed37deef4341a2c37fa0bbb3bac1",
            "a7e4a4dfb7054ca8bfcf7358999ae34c",
            "017e9f5721174b169f12cff28faee34d",
            "9be4c5e49284455fbc1afb8a360de1b0",
            "e4fc0ef10700445ba9a49390c45a5716",
            "5abb6e6728184cdb9450ed543aea8936",
            "4d231168533a4bf4a63ffbc0d53271cc",
            "920ccb52e17a4a18a329bb756c9c35b3",
            "1e62c62899b8448f8969500f0db73d92",
            "eaeb678c01ef4c89b31db675a724d377",
            "516d0dc1bfc34d0392930e884fac47ae",
            "2973c0914c2c409585aa45ac477d8f5c",
            "27654c501e7a4484805ecffdafdcd185",
            "79ddbccc453643689f129740e5148e24",
            "22e8d2d6a5f1402fa49a849521d137b1",
            "7060341f58854fbf8621a0aef79b4f0c",
            "f481c31c5e0d493c943d489409178d11",
            "3945de5d21e24d8d8f1e9f02d596b46c",
            "117ee29d11ff46d3a77d07d1676e64a1",
            "6ba212cc0c384d3b88498ebc5fe7b60d",
            "efefa88ed4c64045b05db81984e8a3e7",
            "c22dbe3bce3d44f397ab8a427609c1e0",
            "78ce2352d3e14733bb5459a59d94f2bb",
            "7ded086d37894a9e94142d6775d78b91",
            "15c65deba3fa46ea914c95f86dbc01eb",
            "afc89fd0f8f2469a8e787389fb8896b0",
            "0aa649b5e9b2461093c9958fc03f0af8",
            "45dd2cc789e24b958c9055248889db41",
            "f3827cc78cd149408fcb4a38eb7794f6",
            "0280590b0bfe4e8390f172a0f7ea3f12",
            "d4b15a4562e2489c8509f7b45b167575",
            "ab991f7fb1504bd5b6fca2c8357e4c83",
            "2e4d7e53bea847819dd5b72b49a80e9e",
            "f4bcca5b6b7643c99a520db52b769784",
            "44f12b979f684345995cb8694ac2f30f",
            "52b2e86553c14b148d37d3dddc3bc464",
            "d282ec102d7340b897516669788ed66a",
            "d58b2c69a73f406d9115f23d302e2ef9",
            "f99308f66ebd4ee38efc60edc7dd2899",
            "7c4822defba54fe3954f7dadfe04b000",
            "145203b1303a40f6beb445c853de4a7d",
            "8ae31dae07bf40e1914b4fb238d65716",
            "1b1634fa3d4a4faa80555ee6f1672600",
            "b03b50388e9d4327ba761b26b4bb113d",
            "5a22f62bc57c4c78840c1ef51c621996",
            "adbd936819104e65a5a6e94817f928f0",
            "7501aa4a0a5f43f49d6dc10857e33317",
            "95035ec41d6d4a1489b02a4967d2c398",
            "2e9fd37ed5e143fab800eb002c8da795",
            "060c7ff1420142b09b701ea4d9010e55",
            "a916f7891d984a0d9b77cd53f31b7e67",
            "ae7b293071124a9abe156fe6067678d3",
            "c3e2364139bc48379e2089a5c9c475be",
            "69d1a3bfc69e48a986a1f53ea68ffdd3",
            "a5b9bfd0475045f885fb0f77accb9952",
            "bf728e3f4516450aa5bf3c928d188d88",
            "7768ac1f0f97461bafe3db188fe0c2a7",
            "c4e3a3c37ff3462ca9251f86efe35fd5",
            "289f94db4e314211aad1ac6a6be977f6",
            "b0ed87168a41425ab3e3ce9c19d5f6cd",
            "0c622cfb456f4317a2952fb781738e80",
            "e21b6208bbc24dfaa77bbc50637bc2ab",
            "d77b2877257b4b379f74165434212dd4",
            "0b8adcc5ad1148ba847f3a0eb07ff354",
            "8100412933fc4f67a8413b2c7a5f9f9f",
            "7c115c8d0b0142febe517f91d8b91f16",
            "b901332db22e401e905cdef9f9083a7f",
            "2fd02c7a887e40829d1bb16304a14b22"
          ]
        },
        "id": "INW7UEhC-I6b",
        "outputId": "7f22b72b-0c63-4152-f6c2-01da36af2824"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "660b81766f7044b6a1ce2f8c2c45f461",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/30.7k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "017e9f5721174b169f12cff28faee34d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/38.7k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "79ddbccc453643689f129740e5148e24",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/14.8k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "15c65deba3fa46ea914c95f86dbc01eb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/44.0k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "52b2e86553c14b148d37d3dddc3bc464",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split:   0%|          | 0/400 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7501aa4a0a5f43f49d6dc10857e33317",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c4e3a3c37ff3462ca9251f86efe35fd5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating test split:   0%|          | 0/500 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'train': 400, 'validation': 100, 'test': 500}"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "from adapters.composition import Stack\n",
        "\n",
        "dataset_en = load_dataset(\"super_glue\", \"copa\")\n",
        "dataset_en.num_rows"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "epiKaEz5dDVe"
      },
      "source": [
        "Every dataset sample has a premise, a question and two possible answer choices:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:25:09.509817800Z",
          "start_time": "2023-08-17T09:25:09.495856Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ifu4q5IJ-hYI",
        "outputId": "183f3b9d-a16f-45c8-f806-8ed5cee9eacc"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'premise': Value(dtype='string', id=None),\n",
              " 'choice1': Value(dtype='string', id=None),\n",
              " 'choice2': Value(dtype='string', id=None),\n",
              " 'question': Value(dtype='string', id=None),\n",
              " 'idx': Value(dtype='int32', id=None),\n",
              " 'label': ClassLabel(names=['choice1', 'choice2'], id=None)}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_en['train'].features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uVa3Vk0QdNYI"
      },
      "source": [
        "In this example, we model COPA as a multiple-choice task with two choices. Thus, we encode the premise and question as well as both choices as one input to our `xlm-roberta-base` model. Using `dataset.map()`, we can pass the full dataset through the tokenizer in batches:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:25:29.453443600Z",
          "start_time": "2023-08-17T09:25:21.211501700Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 209,
          "referenced_widgets": [
            "9a088b764a214b95a00fac01005eb726",
            "9620833757c54c9fb1b1216c40372459",
            "97392a3c7c244bacb931a81947acca06",
            "57c0e7dd3f644dfa8cbb34e79ee695a3",
            "83b01f9793e540e6bcf40f3c65f5359f",
            "495b95595ddd4f15997ea0e6b573fde5",
            "faed48642f37415ea41a4f6155311666",
            "811085599e73470c9cc3b228f0cf2ce4",
            "6e1f35bea40044e5864d0b14c402bd9e",
            "cb44cb18f2904a4a8379b42f4597b591",
            "effcf408a606435286f6b74b52d127c7",
            "bf7d06fbfb7d45c98dde0900cebc16a3",
            "8d6236259648404b99bd6ea9e19d7c48",
            "cea20d44ab3b408c8b41c2a475d75113",
            "19a6a0e0d1af471b8066ad37f12475e7",
            "74accac847e04f60ae59ca1367c59b71",
            "b29700bab9bc499dbbeba804bccc2cce",
            "3c36d64114264fcb82a159f580585096",
            "1d634f5a0ee34c3f8ca0edb228bc9ed9",
            "c96d20da080847cdb20534fbf761b5b6",
            "87c4fc3dca7f433ca231128105a54666",
            "378ee4cd7535482289464c78fdfccff5",
            "785aa76f50cb44cea93bc8d26c8fe117",
            "55350a29295a4e049f8c386412667a1c",
            "11717755bc314c7fa47765a8042a5bc5",
            "0f07f12c5f8a4e158aed6db3b6a5c719",
            "7d6e7b8946c4475bb45c4b8635eaedb8",
            "ffe9e9464031467c93450fb974f9a3b3",
            "97638ea222754090ba4ca37063de36c6",
            "0cb39e011e5b4373a25d2e4d0f158b30",
            "a6e4ccabfd9f42d99d66848938467f2d",
            "ef81ac8f601d43dc8dc9f63e86ce6d5e",
            "761012fd1a874580b2292b344e90aa36",
            "5221149fd81940099402d651f8dce8d5",
            "b1897978affa4617b5d58c3ce0ef5e44",
            "ed690821c13c49df9c559d5ae1240fd6",
            "623a5955ba01446c837cb851465a1c2a",
            "ac4fe7baf6524bd1902212b1a71cbbdf",
            "a1d0d1e41daf4862a5f5f1d681fc009c",
            "8fa6ba880e6342deb538327b05151f78",
            "6b303cc91755460190a5ae0466d82d1a",
            "5992e3e857704d62a5ceb8135ed90f78",
            "500d52783fdd4008a5e02b5439bcd5f0",
            "9397f9387c6d4edf8e0296a26da23db7",
            "a1aa441180e7414bbd30b297a9105643",
            "5d2e7a7f5e6d43d38e5d9a839bad6618",
            "b8b21b05ffca4ddba8e0e4256b79e46d",
            "a9fb5ea6bd2f4a4d94a120cd67a952c6",
            "452e52e81511403db6036d7f610eaa7f",
            "a8fb3390bc7e4f489a1eed8855249780",
            "b82099eefb964f83bc2a99ecaf6b21d8",
            "3d013291a7df403f8c9b30e2beffedb8",
            "5a6333761660446084db896cf8fa3aed",
            "0efe23c68b3d4b9b92c5f89648656910",
            "4aa1ebeecbba4de59fe0104150245fae",
            "fe314b1f2e5d4671bb820bfe16c197e6",
            "8707b73e0a354d0a965cde671dbd7404",
            "ff1a5195281d45ef80af3773331cef74",
            "fc53327bb651457eaaef2bfd17888253",
            "37b69881a6aa453bb5aca22fa5364c23",
            "ca6f26f9b3dc40309fd6f9ee9ead26bb",
            "ad10e12012414b5695cc6977ffd6a9be",
            "5f4b4f5191754b4eb8aebd5af6c7942d",
            "6e6e76fa032f4e579653bf46846f885f",
            "95d531c8744d4a71b228cd0177b72568",
            "2e9e85dcb24941468f9a1de54f613ce0"
          ]
        },
        "id": "hEnRCQfE_Oi3",
        "outputId": "99c4bf2b-8552-4a8c-9530-67930c4a187d"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9a088b764a214b95a00fac01005eb726",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/615 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bf7d06fbfb7d45c98dde0900cebc16a3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)tencepiece.bpe.model:   0%|          | 0.00/5.07M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "785aa76f50cb44cea93bc8d26c8fe117",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/9.10M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5221149fd81940099402d651f8dce8d5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/400 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a1aa441180e7414bbd30b297a9105643",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fe314b1f2e5d4671bb820bfe16c197e6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/500 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"xlm-roberta-base\")\n",
        "\n",
        "def encode_batch(examples):\n",
        "  \"\"\"Encodes a batch of input data using the model tokenizer.\"\"\"\n",
        "  all_encoded = {\"input_ids\": [], \"attention_mask\": []}\n",
        "  # Iterate through all examples in this batch\n",
        "  for premise, question, choice1, choice2 in zip(examples[\"premise\"], examples[\"question\"], examples[\"choice1\"], examples[\"choice2\"]):\n",
        "    sentences_a = [premise + \" \" + question for _ in range(2)]\n",
        "    # Both answer choices are passed in an array according to the format needed for the multiple-choice prediction head\n",
        "    sentences_b = [choice1, choice2]\n",
        "    encoded = tokenizer(\n",
        "        sentences_a,\n",
        "        sentences_b,\n",
        "        max_length=60,\n",
        "        truncation=True,\n",
        "        padding=\"max_length\",\n",
        "    )\n",
        "    all_encoded[\"input_ids\"].append(encoded[\"input_ids\"])\n",
        "    all_encoded[\"attention_mask\"].append(encoded[\"attention_mask\"])\n",
        "  return all_encoded\n",
        "\n",
        "def preprocess_dataset(dataset):\n",
        "  # Encode the input data\n",
        "  dataset = dataset.map(encode_batch, batched=True)\n",
        "  # The transformers model expects the target class column to be named \"labels\"\n",
        "  dataset = dataset.rename_column(\"label\", \"labels\")\n",
        "  # Transform to pytorch tensors and only output the required columns\n",
        "  dataset.set_format(columns=[\"input_ids\", \"attention_mask\", \"labels\"])\n",
        "  return dataset\n",
        "\n",
        "dataset_en = preprocess_dataset(dataset_en)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xs21MzEQ_0v4"
      },
      "source": [
        "## Task Adapter Training\n",
        "\n",
        "In this section, we will train the task adapter on the English COPA dataset. We use a pre-trained XLM-R model from HuggingFace and instantiate our model using `AutoAdapterModel`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:31:35.849814100Z",
          "start_time": "2023-08-17T09:25:45.287070400Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "421306178c18489abcdb27c1265ba6aa",
            "f47ee50ce7bd4024b15381bcf4ed0213",
            "3153dae8974246a8ba3217a47ce8ee7c",
            "f4fc3e122b954a949448b09630857ec0",
            "6ae390407899480292d8a0ec2e245bae",
            "807ae90cbea448d5aa02c4e9418acdaf",
            "9aa53ebe612b42c5a29ca0fc32a28c71",
            "6fae67869ca04721bd03f4995c3259d6",
            "dfb749a8532d422f87e9c8bd4302a4a0",
            "c713c1586a344d3da06b0d87764b4afb",
            "6ff76b8372584d09a7df7bb52fccd176"
          ]
        },
        "id": "fnq8n_KP_3aX",
        "outputId": "9efa442e-b67d-4b98-a5b1-14dc4e3542b7"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "421306178c18489abcdb27c1265ba6aa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading model.safetensors:   0%|          | 0.00/1.12G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from adapters import AutoAdapterModel\n",
        "from transformers import AutoConfig\n",
        "\n",
        "config = AutoConfig.from_pretrained(\n",
        "    \"xlm-roberta-base\",\n",
        ")\n",
        "model = AutoAdapterModel.from_pretrained(\n",
        "    \"xlm-roberta-base\",\n",
        "    config=config,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BDCNJzTXezcn"
      },
      "source": [
        "Now we only need to set up the adapters. As described, we need two language adapters (which are assumed to be pre-trained in this example) and a task adapter (which will be trained in a few moments).\n",
        "\n",
        "First, we load both the language adapters for our source language English (`\"en\"`) and our target language Chinese (`\"zh\"`) from the Hub. Then we add a new task adapter (`\"copa\"`) for our target task.\n",
        "\n",
        "Finally, we add a multiple-choice head with the same name as our task adapter on top."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:33:12.495169Z",
          "start_time": "2023-08-17T09:32:53.909908500Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 113,
          "referenced_widgets": [
            "d8492798e90e4e1c9d52dad2abfff1a2",
            "236e4fb59b3e418993ed329310de2457",
            "868cc846c7684641bc1c25f491c3d2b0",
            "72c0d49772d9407db838009434d8caa6",
            "07ff9bc229874e51b085d015bb1c433e",
            "5f96be3d44ed4d37b73dcc358a5756be",
            "355075984c274acba555a874dc51f987",
            "0243ae5e50fb4186989be9966c9962c4",
            "6cfed16922c443988ab647765ebbd801",
            "3ec325fb22fe42df8d6564ca93a66bc2",
            "1a555ff033ce47f389abc112b42475e9",
            "ee1dd3ac0e364a9d81211fb48e3a63c2",
            "a6c03bbb26874ef786886e3874b96f45",
            "ed31165ad7b943d8a48a0d82d99627c8",
            "8a3dbcf2432e4473acffdc3cd2d7c287",
            "2f5bd7a50398424da94c61c6f8e3a125",
            "6c27af5311e247a3976cdf5b03c0b717",
            "1f47605b01224bcfa8464c6d498fa1d1",
            "f8805f4f95ab43cca5aa05fc81eaba06",
            "6f88804f321b4e2d81ee387fd7299799",
            "afa0e74f7eb24520863fe56ad5ac4ef3",
            "1883a118e9ab4e8b96e0941113d9c240",
            "a1c8238ae1e441e9a525141b41997c1c",
            "d4e33eaaad674df0ac7483d68aa46033",
            "30a08ac6bd9445508d6fd68ca162bd49",
            "4f144677ebd54a87a6b493a8be5486c0",
            "9cf949851039440990727c62b9710e6b",
            "2ac898ff598e40a1a28ef7b5350f3f57",
            "7b0315b98dd74398b562b753b00703da",
            "80ebeea5e6ba46a0a228965232996340",
            "398722250162461dbd886d13a8e93257",
            "3b26ab59e14a4b4c891c7bcb50cf495a",
            "cda3c917778848c98dbd7752d798989b"
          ]
        },
        "id": "jRqbBgS0BoHJ",
        "outputId": "7358bb8e-55f8-4c84-9d20-d901290c7838"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d8492798e90e4e1c9d52dad2abfff1a2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)lm-roberta-base.json:   0%|          | 0.00/609 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ee1dd3ac0e364a9d81211fb48e3a63c2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)eiffer/en_relu_2.zip:   0%|          | 0.00/29.6M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a1c8238ae1e441e9a525141b41997c1c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (…)eiffer/zh_relu_2.zip:   0%|          | 0.00/29.6M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from adapters import AdapterConfig\n",
        "\n",
        "# Load the language adapters\n",
        "lang_adapter_config = AdapterConfig.load(\"seq_bn\", reduction_factor=2)\n",
        "model.load_adapter(\"en/wiki@ukp\", config=lang_adapter_config)\n",
        "model.load_adapter(\"zh/wiki@ukp\", config=lang_adapter_config)\n",
        "\n",
        "# Add a new task adapter\n",
        "model.add_adapter(\"copa\")\n",
        "\n",
        "# Add a classification head for our target task\n",
        "model.add_multiple_choice_head(\"copa\", num_choices=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTGQRNSd0Gmt"
      },
      "source": [
        "Using `train_adapter()`, we tell our model to only train the task adapter in the following. This call will freeze the weights of the pre-trained model and the weights of the language adapters to prevent them from further finetuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:33:22.096474100Z",
          "start_time": "2023-08-17T09:33:22.074532300Z"
        },
        "id": "7wBpjGWZ4v7O"
      },
      "outputs": [],
      "source": [
        "model.train_adapter(\"copa\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "60qIas8-il92"
      },
      "source": [
        "We want the task adapter to be stacked on top of the language adapter, so we have to tell our model to use this setup via the `active_adapters` property.\n",
        "\n",
        "A stack of adapters is represented by the `Stack` class, which takes the names of the adapters to be stacked as arguments.\n",
        "Of course, there are various other possibilities to compose adapters beyond stacking. Learn more about those [in our documentation](https://docs.adapterhub.ml/adapter_composition.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:33:24.908948500Z",
          "start_time": "2023-08-17T09:33:24.892991800Z"
        },
        "id": "zgGqHJQbijgg"
      },
      "outputs": [],
      "source": [
        "# Unfreeze and activate stack setup\n",
        "model.active_adapters = Stack(\"en\", \"copa\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBDvwmUf_-mc"
      },
      "source": [
        "Great! Now, the input will be passed through the English language adapter first and the COPA task adapter second in every forward pass."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fb8BY5RAmzkd"
      },
      "source": [
        "For training, we make use of the `AdapterTrainer` class built-in into `adapters`. We configure the training process using a `TrainingArguments` object.\n",
        "\n",
        "As the dataset splits of English COPA in the SuperGLUE are slightly different, we train on both the train and validation split of the dataset. Later, we will evaluate on the test split of XCOPA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:33:43.542080300Z",
          "start_time": "2023-08-17T09:33:42.546744700Z"
        },
        "id": "j0gFxQRdDkQ6"
      },
      "outputs": [],
      "source": [
        "from adapters import AdapterTrainer\n",
        "from transformers import TrainingArguments\n",
        "from datasets import concatenate_datasets\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "    learning_rate=1e-4,\n",
        "    num_train_epochs=8,\n",
        "    per_device_train_batch_size=32,\n",
        "    per_device_eval_batch_size=32,\n",
        "    logging_steps=100,\n",
        "    output_dir=\"./training_output\",\n",
        "    overwrite_output_dir=True,\n",
        "    # The next line is important to ensure the dataset labels are properly passed to the model\n",
        "    remove_unused_columns=False,\n",
        ")\n",
        "\n",
        "train_dataset = concatenate_datasets([dataset_en[\"train\"], dataset_en[\"validation\"]])\n",
        "\n",
        "trainer = AdapterTrainer(\n",
        "    model=model,\n",
        "    args=training_args,\n",
        "    train_dataset=train_dataset,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iKlYjA9rm2Kp"
      },
      "source": [
        "Start the training 🚀 (this will take a while)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "id": "vSUs2FjXDmsx",
        "outputId": "bb310f23-94d0-48c2-b10d-e0356340cbd6"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='128' max='128' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [128/128 00:50, Epoch 8/8]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>100</td>\n",
              "      <td>0.697100</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=128, training_loss=0.6968793570995331, metrics={'train_runtime': 55.5306, 'train_samples_per_second': 72.032, 'train_steps_per_second': 2.305, 'total_flos': 293495135040000.0, 'train_loss': 0.6968793570995331, 'epoch': 8.0})"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axvDsmnJnGUG"
      },
      "source": [
        "## Cross-lingual transfer\n",
        "\n",
        "With the model and all adapters trained and ready, we can come to the cross-lingual transfer step here. We will evaluate our setup on the Chinese split of the XCOPA dataset.\n",
        "Therefore, we'll first download the data and preprocess it using the same method as the English dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:34:08.620963600Z",
          "start_time": "2023-08-17T09:34:04.407241200Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426,
          "referenced_widgets": [
            "216fc1dd69b840788de0425040d646a3",
            "9ab974ee666c47b0ac46d5112d5abdf5",
            "e500b37792734768892e8548250ca1a5",
            "24e3f9dc7a9742eaafad98f423a550df",
            "c475cfa85927479b9604bb4a33183957",
            "afe95ec0dfc043f5ab02bb91f224ae4a",
            "006b4341bbea4f29a9f5c053045df606",
            "c47290facbb34126bf648a58041b7110",
            "2da273f7585c4b45a36f55d5f112e581",
            "4af55acfc6274c88b14c8441dd395df2",
            "fcd662335f9447cb87bd66b55fea1f41",
            "1937074f179040e2b96d7972dc1a6998",
            "cb4f9dccc08c4caab16774ea9a133b2b",
            "7917c44d3ec94fc7ad80e1657fc1627b",
            "11b11def7bff484191999b63753e780b",
            "7e23d208fd96406186ea49ddc340eaa0",
            "8b2ef4dfe5024e339037b0dad137c689",
            "d99b8bb92a4c450caffc46a8a132c986",
            "72d73ed65b4b4f30a80b21cedc8cf3e1",
            "65090b4bf9a84377adaf0b965a42d254",
            "ee1a05c0abb94b3fabbbbcbbc44833b8",
            "740f9bd6524f4af0bc79be53039a7961",
            "8b2b2f05acac4711a1e31018247dda4b",
            "4ee4a536f1ae46529aa5cc6688bc5f2a",
            "cd606aea6d57434398ff3202414c33f0",
            "9fc643b1837342a18126cacc88b6b46d",
            "7e550ac5e8c5475e875eb58a9adfac56",
            "327bfa1cb0e54c95993ce8fe01be7abc",
            "e20cfd11370548d284ae8b6530911e0d",
            "d06d4b63f0a84e9d8831d8f6f0d773ee",
            "1a5d8044a2254d0881d854c0b01d7141",
            "7d0e3446543f4eb7ac827c94c0947f2a",
            "6557f70fcb2e475495dc88ebec7e6dd8",
            "1759469306854432b2d4d6618fc45776",
            "cf257fbfb5824f99af08146161fb0f67",
            "03cb547ad56b4c1db3fb373969eacf85",
            "f2619549195c4090a453c551710cdb65",
            "988ea2ed503f44b5bc8ff16844967d48",
            "23e490b5be9d45f682030f7193b740fc",
            "7006b0c1017a411e97b8a4d203a4b046",
            "b2c3e773ab524ad381102340a59c7e9f",
            "11e1cbaec2fb4276b7e0f53465f214ee",
            "240b1eb858834defb5f1069f017e859e",
            "26b04e0848d74b20988237710bb52aa6",
            "40c2e934eb384c8c8d3f19a9779b28d9",
            "8b3788c01be44040893f782e6aa12e7b",
            "3501ed620e24462ebe622aceabb3bd3d",
            "ded200a20be94aea92af5d408156fcca",
            "06db73f59d764ac2b52cec52fdcd1d04",
            "5ecc0a746abb44cdb1566203edb307f4",
            "1bb95488e26d4cc9865b91d1af06e792",
            "73bcd540ea0242c78f8fdddb348fff3e",
            "1014511855324a61a1c90e299470f4b9",
            "1eabfe440e39437190c7a466b0967798",
            "63aa346b6a1b491183dc78bfd5127ea3",
            "650cc1472a394fdc854f7fcac4594317",
            "25962637853646f0a19e320bee136091",
            "387711b2140f4b9ebbcb363e1067d6f7",
            "cd09018d44c34714aeb6e90b19729303",
            "b5c42155aeb843b98172b25e79cbc811",
            "ddd3c45654f4476eb1eb2d30e621f6a3",
            "bae3fdd732764d08a5e417f76a94b17d",
            "4f63dba0f0df41748b523ada60679496",
            "5994264c53e54ca5b50a06cd8ef2556b",
            "ed61eabf01c5432a90c3de5c67039746",
            "c760f0df4bbd49f9a7dda28bb0ac3482",
            "7f9ef06955b74b1d95011fe7c38c4cb2",
            "d429427bc4804c96bf1dab8b52d3fef1",
            "cf19ebd836b24130b22bb9d2290d59a3",
            "0f4c24c2eaa84981a51ca8043e881218",
            "7bc54b222b06474196c44e429ffb0f05",
            "86e49105023c4ed99f80342f2d48fbc7",
            "d0ea922e12434bdfbb500d6c883589bf",
            "f9b6e4363f1b4e3392d1da9f3814f9cc",
            "901698bb5b8f4a3ea00848f0f2b5046c",
            "99360f3c8571451f958c1dd73283674e",
            "1cc3cec507844485850a5b95d3db6e9d",
            "66881711810a4ad28f08bff9bd15e6fd",
            "2dcbf0d434494c4ca36c74fb97be7df5",
            "8e6c82700dc74db3bd6d3946f93c956c",
            "568e8a9bb41841a99c608485e4a67ec8",
            "af6b21dc6f3f48c7ae5c240b403b3c91",
            "befb2622d31b436f8d8cb25d35243989",
            "26b4f219a8ef4d52a4c34ee304e31fd1",
            "907db8b968a64827a2f4783657949128",
            "edeb97603ab34a499516c6589a04a711",
            "053917bbf7fc43a0887752403a972737",
            "221597e24fd14a0e8e67321692f615b8",
            "6e64c9f038fa4e57a374802726b5db3d",
            "af0166beb7444cab8f012a8b4df18475",
            "dae6332c719c479c91156d8651c3c59f",
            "370bb26be7b34dc181ea88756f99afa3",
            "57853bbe0e9942cf8c3e73778f8ff2fc",
            "ac877b5528c744b7b841c17b7516b879",
            "9ad5687370ed4164a905809751571663",
            "ceaa596549fd423189786aff7d865a99",
            "ee67317d933c47d48d3e46cb52e02324",
            "2fbc66f9f17043be8b03fe8cb84ac0d6",
            "24e6ff03c6b04b94bd729a09a93d4841",
            "3bf4f93b8e7e4b4aa8d4af2fd8683a56",
            "764a4f006ad3411680dc35ab7f77f4d0",
            "75f861994a384edcba5991e24ec0acb2",
            "81f029f951bf4dd4bbbc7ee6e4cc4d9e",
            "e9e86e5e4a1341d592f72110112f276d",
            "17b4e1697bb848ad9b1c74993a05b64d",
            "b68f616e1ec54acbbd6c70f2ee9282cd",
            "b36a3e769ea946d9b3a2eee4145d858a",
            "bbc6b3b9564940098f971fc7c711d61b",
            "6c94041132c241b19e0b0d697b044b58",
            "2bc1eb041c624197b006ca3c29e6819d"
          ]
        },
        "id": "cMgM1supdxpw",
        "outputId": "e7dbf1f2-60f8-480d-8cd6-b601cecd697c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2080: FutureWarning: 'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0.\n",
            "You can remove this warning by passing 'verification_mode=no_checks' instead.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "216fc1dd69b840788de0425040d646a3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading builder script:   0%|          | 0.00/3.90k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1937074f179040e2b96d7972dc1a6998",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/60.6k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8b2b2f05acac4711a1e31018247dda4b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/19.0k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1759469306854432b2d4d6618fc45776",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "40c2e934eb384c8c8d3f19a9779b28d9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/5.32k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "650cc1472a394fdc854f7fcac4594317",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/23.3k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7f9ef06955b74b1d95011fe7c38c4cb2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "66881711810a4ad28f08bff9bd15e6fd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating test split:   0%|          | 0/500 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6e64c9f038fa4e57a374802726b5db3d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3bf4f93b8e7e4b4aa8d4af2fd8683a56",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/500 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'labels': 0, 'input_ids': [[0, 6, 4941, 55359, 1173, 7825, 30638, 90132, 44507, 5702, 1562, 30, 22304, 2, 2, 6, 3800, 2165, 15068, 69175, 30, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [0, 6, 4941, 55359, 1173, 7825, 30638, 90132, 44507, 5702, 1562, 30, 22304, 2, 2, 6, 3800, 2165, 1128, 30, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]}\n"
          ]
        }
      ],
      "source": [
        "dataset_zh = load_dataset(\"xcopa\", \"zh\", ignore_verifications=True)\n",
        "dataset_zh = preprocess_dataset(dataset_zh)\n",
        "print(dataset_zh[\"test\"][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_0w3ofGZC4Oi"
      },
      "source": [
        "Next, let's adapt our setup to the new language. We simply replace the English language adapter with the Chinese language adapter we already loaded previously. The task adapter we just trained is kept. Again, we set this architecture using `active_adapters`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:34:44.463042Z",
          "start_time": "2023-08-17T09:34:44.446088100Z"
        },
        "id": "V04UntKfeK_z"
      },
      "outputs": [],
      "source": [
        "model.active_adapters = Stack(\"zh\", \"copa\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F9RAmZLzDk-j"
      },
      "source": [
        "Finally, let's see how well our adapter setup performs on the new language. We measure the zero-shot accuracy on the test split of the target language dataset. Evaluation is also performed using the built-in `Trainer` class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2023-08-17T09:35:01.032697600Z",
          "start_time": "2023-08-17T09:34:47.053111900Z"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "CrW1cJyaeMox",
        "outputId": "2a548b3f-93e8-46b2-eb09-60ff63ba63b9"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='63' max='63' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [63/63 00:04]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'eval_loss': 0.6925755739212036,\n",
              " 'eval_acc': 0.518,\n",
              " 'eval_runtime': 4.9675,\n",
              " 'eval_samples_per_second': 100.654,\n",
              " 'eval_steps_per_second': 12.682}"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "from transformers import EvalPrediction\n",
        "\n",
        "def compute_accuracy(p: EvalPrediction):\n",
        "  preds = np.argmax(p.predictions, axis=1)\n",
        "  return {\"acc\": (preds == p.label_ids).mean()}\n",
        "eval_trainer = AdapterTrainer(\n",
        "    model=model,\n",
        "    args=TrainingArguments(output_dir=\"./eval_output\", remove_unused_columns=False,),\n",
        "    eval_dataset=dataset_zh[\"test\"],\n",
        "    compute_metrics=compute_accuracy,\n",
        ")\n",
        "eval_trainer.evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "op8M7AfYnWhs"
      },
      "source": [
        "You should get an overall accuracy of about 56 which is on-par with full finetuning on COPA only but below the state-of-the-art which is sequentially finetuned on an additional dataset before finetuning on COPA.\n",
        "\n",
        "For results on different languages and a sequential finetuning setup which yields better results, make sure to check out [the MAD-X paper](https://arxiv.org/pdf/2005.00052.pdf)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.6"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "metadata": {
          "collapsed": false
        },
        "source": []
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
